OnlineHost: The auditorium consists of two major areas: the audience, where you are right now, and the stage, where the speakers appear. Text which you type onscreen shows only to those in your row, prefaced by the row number in parentheses, such as (2) if you are in row 2. To interact with the speaker, use the Interact icon on your screen. To send your question to the speaker, click on the Interact icon, then use the Ask a Question option.
CJ Dox: Good evening all, and welcome to Pro's Corner. I'm Dox Doxiadis of the Internet Connection. With me is Rich Wiggins of Pro's Corner. Tonight we will be talking about "Index Spamming and other Web Spider Hazards". Now Rich, before we say anything else...what IS a Web Spider and why is "Index Spamming" such a hazard?
RichWig: Well, Dox, I'm glad you asked. Our story begins in the Physics-Astronomy library at a large midwestern university. Our heroine, the librarian, was trying to find out what new cool resources had sprung up on the Net in the area of astronomy. So she innocently connected to a Web index service -- let's assume it was Lycos (because it was ) and she innocently typed "astronomy".
RichWig: Dutifully Lycos returned a list of resources and at the top of the list was a promising page from a technical high school in New York City... So she visited this promising Web site, and upon seeing the page, a gasp escaped from her lips because this page had nothing to do with astronomy!
CJ Dox: So how did it get on Lycos?
RichWig: Shocking, isn't it! Those callow kids at the NYC school had fooled our hapless librarian friend! Weeks earlier, they'd built a page on their Web collection whose title included the word Astronomy, and internally, the Web page had lots of HTML comments that said astronomy astronomy etc.
RichWig: Lycos is an example of a popular Web index, and its "spider", which crawls across the Web looking for new and updated resources, found the page, and it in turn put that word in the index, with a HUGE word count. Now why did these kids do this? What glory can there be in fooling the index, and in misleading our hapless heroine???
RichWig: Who knows, but they've just performed the act of "index spamming"
CJ Dox: Well, kids will be kids, but perhaps an unscrupulous operator of a commercial page would do that too?
RichWig: You betcha! This past week, with great fanfare, General Motors announced its site on the Web. Now a GM dealership might feel motivated to overload a page with "GM" and "General Motors" and related terms, and a lowly dealership will appear more prominently on screen than the General itself. And the funny thing is, there's no written rules about this practice, no acceptable use policies against it.
CJ Dox: Rich, for the benefit of those who are new to the phenomenon: in the "astronomy" case, wouldn't the word in question appear all over the page and thus look really ridiculous?
RichWig: Not necessarily. One way to pull this scam is to put the word inside HTML comments. These do not appear on screen, unless of course you invoke View Document Source. Now, you might say "Well, then, spiders should just ignore the text in comments!" But comments can legitimately be used to include keywords that people might search for. In fact, is anyone in the audience aware of how another index spammer used the keywords-in-comments trick to embarrass a Net full of people looking for salacious material on the Net???
CJ Dox: If you are aware, hit the Interact With Host button and enter it as a comment.
CJ Dox: (If you have a question on tonight's topic, hit Interact with Host and enter a question.)
RichWig: Now in the meantime the people who invented Web spiders, who are smarter than the average bear, are working on coping with this problem. I interviewed Dr. Michael "Fuzzy" Mauldin, who invented Lycos, a couple of months ago. He uses the term "spamdexing" instead of "index spamming" and he says "If I catch you spamdexing that's the fastest way to guarantee your URL will never be indexed in Lycos again".
RichWig: Of course, Lycos has 20M URLs in the index, so Fuzzy would be mighty busy manually inspecting for spam!
CJ Dox: Mr Diddum asks...
Question: So, relative position in an index list depends on total word count in readable text AND comments?
RichWig: Depends on the index, but yes, that's true for at least some of them. Different indexers have different ways of handling proximity of words to one another, which is another factor. Some indexes, like Alta Vista, allow you to type phrases... So if you type in "A Night at the Opera" you will get different results than if you type those words into the box with no quotes.
CJ Dox: AFL Sarah would like to know...
Question: Rich, can't the spiders be set to ignore MULTIPLE, sequential spamdexing listings?
RichWig: Absolutely! And that's what I was heading towards when I mentioned that Fuzzy Mauldin can't inspect each page for spamming.... So he is training Lycos to do exactly what you just proposed. Of course, the nefarious kids at the high school in NYC could get craftier. They could put up Web pages with the word "astronomy" separated by other words and thus fooling the spam detector.
RichWig: In fact, I can think of an even more dastardly stunt they might pull. Can you? (Hints available) For instance, I was in a meeting at my Day Job the other day, trying to figure out how one large midwestern university is going to comply with the #$(#*@($#*( Communications Decency Act.
RichWig: In the room we had 3 people who brought 3 slightly differently arranged printouts of the Act, downloaded from different places on the Net. You see where I'm headed with this? Sarah, I bet you're a step ahead!
CJ Dox: PossumKat asks...
Question: Rich, do most "spider" web indexes have feedback forms, so users can note if a top-listed site is really a scam or a dog?
RichWig: Yes, most sites do indeed offer feedback, but with the size of the Web today, and with its meteoric growth, there is increasingly little chance that a human will ever see your comments. The Alta Vista people even decided not to offer a "Remove this URL from the index" option because of folks (our friends in NYC?) deleting URLs they do not own.
RichWig: We had a resource listed in the Lycos 250 that was coded as belonging to another university and it took dozens of feedback messages to get 'em to fix it. Alas, these feedback forms get dozens of user queries every day. Instead of "feedback" people type in the search they want satisfied!
CJ Dox: Rich, it would seem that a simple solution would be to only read the "visible" page and not the source. What would be the problem with that?
RichWig: Ahhh, but now we get to the even MORE nefarious, even MORE dastardly kind of spam I was alluding to. What is to stop me from putting up a Web page that has a title of "Communications Decency Act of 1996" (for instance) and putting in it whatever language I want? In the visible text? Or what is there to stop our high school friends from composing completely bogus astronomy articles, made of whole cloth? These indexers are dumb robots. They have no way of separating wheat from chaff.
CJ Dox: For that matter, Rich, a well intentioned spoof may come up as the real thing?
RichWig: True! Even now, go look for the Star Trek Voyager home page using Webcrawler or Lycos. It will take you a LONG time to find the official Paramount page among the dozens of pages put up by rabid Trekkers... Can anyone in the audience think of possible solutions? Send up your ideas by hitting the Interact button.
RichWig: To put the problem in perspective, if the Web consists of 20 million URLs, and if it's growing by a factor of 7 each year, then within a couple of years we're looking at 1 BILLION URLs in the Web. Here are some ideas as to how indexers could be smarter: Take the problem of the Paramount page. If you inspect its URL, it looks a little different than the typical "personal" page on the Web. An indexer could give greater weight to a URL that looks like www.paramount.com as opposed to www.msu.edu/users/smith .
RichWig: It could even give lower weight to pages that have tildes in them, which is a sign on many Web sites that a page is a personal page. It could weight pages more heavily based on how often they are visited as a result of OTHERS' searches. And it could lend more "credence" to pages that are pointed to by thousands of links found elsewhere on the Web.
CJ Dox: Deimos1 suggests something similar...
Question: What about someone designing a search engine that only has official home pages?
CJ Dox: That is, I think, a restricted engine.
RichWig: That's certainly another possibility. And in fact, I predict we will see that. There are two ways we could get there: The Yahoo people could implement a spider, or someone with a spider service (say, Webcrawler or Lycos) could build a Yahoo-like list of starting points, and set the spiders crawling within those specific subject areas. Indexing only lists of known sites.
CJ Dox: As a follow up comment, Deimos1 notes...
Comment: But the URLs' have to be submitted to this search engine.
RichWig: Yup, and we're back into manual intervention. We face this issue at the university where I work. We want to build two indexes: one of "official" university pages, and one of personal pages. We're learning that in some departments there is no heuristic to determine which category a given page lies in! I guess we have to train a spider to look for photos of pets and vacation shots and links to the Dilbert home page!!!
CJ Dox: Mr Diddum has an interesting comment...
Comment: I tuned in to find out how to let users know our page is the real magilla and not some spooferz.
RichWig: Aha, you are a Webmaster concerned with demonstrating your own pages' authenticity? I can't think of a perfect solution. You can't stamp each page with a royal seal like kings of old would've done -- your spoofers can steal your seal! If it's a trademark you can legally limit its use by others, of course.
RichWig: Some companies make things more confusing. DEC's popular AltaVista service is located at altavista.digital.com. But you might look for it at altavista.dec.com, because they are inconsistent in their own company's use of domain names! Worse, there is another company at www.altavista.com! Those poor blokes had to put up a new pointer off to DEC when they started getting thousands of extraneous hits per day.
RichWig: Alas, I don't have a good specific answer for you as to how to demonstrate the authenticity of your pages. This would be an EXCELLENT topic for the Pro's Corner message board. Why don't you post it there?
RichWig: Well, folks, I see by the ole clock on the toolbar we're about out of time. Thanks for joining us tonight in Pro's Corner. Come visit us every Tue/Thu at this time, and be sure to pump us with questions and comments on the message boards!
CJ Dox: You can find a transcript of this and all Pro's Corner conferences at keyword PROS CORNER in a few days.
